1 I Exoress Mail I\ahe1 Mn 
2 

3 EF EV^SaQSBH US _ 

4 USE OF TEMPLATES FOR COST-EFFECTIVE SECURE 

5 LINKING OF VIDEO STREAM OBJECTS 

6 Yakov Kamen 

7 Leon Shirman 

8 This patent is a continuation in part of U.S. Patent Applications 09/389,916, 


9 09/390,047 and 09/390 ? 044 3 each filed September 3, 1 999, each of which incorporated 

10 herein by reference. 
11 

12 Background of the Invention 

13 It is known in the art to annotate or "mark" objects in a video stream with an 

14 interactive link. For example, suppose one was watching a video program of an 

15 automobile race, and there were several racing cars shown driving around a track. Also 

16 suppose that these cars were marked with interactive links. This means that one can 

17 position a cursor over the image of one of the cars and click on the image (or 

18 alternatively, click on special icons associated with the car). This causes the system 

19 controlling the video screen to take a predetermined action (e.g. execute a software 

20 program, or initiate some other event such as a telephone call) or display information that 

21 is linked to that image. This link can be in the form of a pop-up window for note 

22 annotation. (A pop-up window for note annotation is a small window that appears on the 

23 screen so that a user can write a small note. It is like an electronic "post-it" note.) 

24 Alternatively, the link can be in the form of a URL. If the link is invoked, the system will 

25 display a web page depicting information concerning the car or driver that has been 


1 clicked on. For example, the system can display details concerning the driver. As 

2 mentioned above, by clicking on the icon or image, one could also initiate an action, e.g. 

3 a telephone call to a merchandise company to buy a memorabilia product related to what 

4 has been clicked on. 

5 During annotation, an area or region within the video frames surrounding the 

6 image of an object of interest (in the above-mentioned example, the racing car) is 

7 established as an "active area." If one moves the cursor into the active area and clicks, 

8 one will initiate an action or invoke the link corresponding to the object within that active 

9 area. Typically, an object associated with the active area moves about the video screen 

10 during a video clip. For example, a video clip of an automobile race shows a group of 

1 1 cars moving about a racetrack, and these cars typically move about a television screen 

12 during the clip as the race proceeds. In such a video clip, a car might move from the right 

1 3 side of the video screen to the left side of the video screen. Thus, the active area must 

14 also move across the television screen to keep up with the object of interest. The above- 

15 incorporated patent applications pertain to methods for annotating video clips, i.e. 

16 defining the active area for each object of interest, and for each frame, in the video clip. 

17 These methods permit annotation with a reduced amount of human involvement in the 

18 process. It would, however, be desirable to further simplify the process of defining active 

19 areas in a video clip. 
20 

21 Summary 

22 A method in accordance with the invention comprises the step of providing a 

23 template or a set of templates to be associated with a video screen. The template defines 

2 


1 a set of active regions of the video screen. The active regions typically do not overlap 

2 one another. Each active region corresponds to a process or link that can be invoked. In 

3 one embodiment, the link is to an internet web page* When that link is invoked, the 

4 contents of the web page are displayed on the screen. 

5 In another embodiment, the link is to a source of information other than an 

6 internet web page. For example, the link can cause a video system to display information 

7 from a memory to be displayed on the screen. The memory can either be local or non- 

8 local. 

9 In yet another embodiment, invoking the link causes a software routine or 

10 application program to be executed. 

1 1 Alternatively, invoking the link initiates an event, e.g. a telephone call 

12 In one embodiment, the active areas of the template are visible to the user. For 

13 example, the active areas of the template can be a particular color, or bear a particular 

14 icon, or can be shaded. Thus, the user knows the location of the active areas, and can 

1 5 move a cursor to the active areas to invoke links associated therewith. In another 


16 embodiment, the active areas are not visible. However, when the cursor is moved to the 

17 active areas, the appearance of the cursor changes, so the user knows that the cursor is 

18 positioned over an active area. In yet another embodiment, the active areas are not 

19 visible, and the cursor does not change appearance when positioned over the active areas. 

20 The video screen can display any type of video information, e.g. a still image, a 

21 video image, a web page, a Windows-type desk top output, or other computer output. 

22 For example, a user can watch a television program on the video screen and manipulate a 

23 cursor to invoke the links associated with the template. Alternatively, the user can read a 
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1 web page on the video screen and can manipulate a cursor to invoke the links associated 

2 with the template, 

3 One advantage of this invention is that the use a set of templates is a simple way 

4 of providing robust dynamic linking. A person using the template does not have to chase 

5 a small moving object on a screen with a cursor in order to invoke a link. 

6 A set of templates in accordance with one embodiment of the imention allows a 

7 content provider to choose the most appropriate template without complicated content- 

8 oriented linking (i.e. without the complication of having the active areas move in sync 

9 with associated objects on the video screen). In some cases this solution can be 

10 preferable to fixed static icon (one fixed template) solutions and completely dynamic 
1 i (moving active area) solutions. 

12 In accordance with another aspect of the invention, templates can be used for 

13 secure communication of data associated with a video stream. For example, in one 

14 embodiment, each active area of a template can be considered as a symbol (or letter) that 

15 is part of an arbitrary complicated password, which allows a user to be re-directed to 

16 selected internet pages, to call a secret telephone number or start a special application. 

17 These and other aspects of the invention are described in greater detail below. 

18 

19 Brief Description of the Drawings 

20 Fig, 1 illustrates a video screen including an overlay template comprising a set of 

21 active areas, 

22 Fig, 2 illustrates a second template overlaying a first template on a video screen. 
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1 Fig. 3 is a block diagram illustrating a video system in accordance with the 

2 invention. 

3 

4 Detailed Description 

5 Referring to Fig. 1 , video system 1 comprises a substantially rectangular video 

6 screen 2 having active regions 4a to 4d and a region 6. Regions 4a to 4d correspond to a 

7 "template." Video screen 2 can display any type of visual information. For example, in 

8 one embodiment, screen 2 displays a still image, e.g. a picture. In another embodiment, 

9 screen 2 displays a movie or television program. In another embodiment, screen 2 

10 displays a web page. In yet another embodiment, screen 2 displays a computer output, 
i 1 e.g. an image of a Windows-type desk top. Screen 2 can be a CRT screen, an LCD 

12 screen, a video projection screen, or other type of screen capable of displaying a visual 

13 image. The image displayed on screen 2 can originate from a conventional television 

14 receiver that receives radio waves or a receiver that receives signals from a cable or 

1 5 optical fiber. Alternatively, the image on screen 2 can originate from the internet, from a 

16 personal computer, a VCR, or other source of visual information, either in digital or 

17 analog form. 

18 In a first embodiment, regions 4a to 4d are not visually distinguishable from the 

19 remainder of screen 2. In other words, if screen 2 is displaying a television program, one 

20 cannot tell, merely by looking at screen 2, where regions 4a to 4d are located. 

2 1 Also shown on screen 2 is a cursor 8. Cursor 8 can be any shape, or have any 

22 appearance. A user manipulates cursor 8 with a control device, e.g. a remote controller 

23 10 comprising a joystick, trackball, mouse, touch pad (e.g. touch pad 1 la) or appropriate 
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1 control buttons lib. If a user moves cursor 8 to region 4a, a link is invoked 

2 corresponding to region 4a. If the user moves cursor 8 to region 4b, a link is invoked 

3 corresponding to region 4b, and so forth. In one embodiment, the link is to a web page. 

4 Moving the cursor to region 4a invokes the link, and information corresponding to the 

5 web page is displayed on screen 2. {In an alternative embodiment, one must move cursor 

6 8 to region 4a and click on region 4a to invoke the link, rather than merely moving cursor 

7 8 to region 4a.) 

8 In another embodiment, the link is to a data source other than a web page. This 

9 data source could be another source of visual information, e.g. another movie or 

10 television program. Alternatively, the link could execute application software. 

1 1 Alternatively, the link could result in the display of data from a local data source, e.g. a 

12 disk drive, or a non-local data source. In one embodiment, the link is to a pop-up 

13 window for note annotation. 

14 As mentioned above, in a first embodiment, regions 4a to 4d are not visible per se. 

15 In a second embodiment, when cursor 8 is moved to one of regions 4a to 4d, cursor 8 

16 changes appearance (e.g. cursor 8 changes shape, color, light intensity, or otherwise 

17 manifests a changed appearance). Thus, a user can determine whether cursor 8 is located 

18 in an active region by observing the appearance of cursor 8. 

19 In a second embodiment, regions 4a to 4d can be detected visually. For example, 

20 regions 4a to 4b can be shaded differently from other portion 6 of screen 2. Thus, a user 

21 can still use the entire video screen to watch a video image, but the user will know where 

22 the active regions are located. In lieu of shading the active regions differently, a set of 
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1 lines such as lines 4a' to 4d' are visible so that a user can see where the active regions are 

2 located. 

3 In a third embodiment, an image is displayed in region 6 of screen 2, but that 

4 image is not displayed in active regions 4a to 4d. Thus, regions 4a to 4d can be used to 

5 display something other than that image, e.g. other visual information such as icons, 

6 alphanumeric information, thumbnail still or video images, and so forth. The images 

7 displayed in regions 4a to 4d can come from any appropriate source, e.g. a cable or 

8 optical fiber, a conventional television receiver, a computer memory (eiti ler local or 

9 remote), a VCR or other source of visional information. 

10 In this embodiment, regions 4a to 4d can be thought of as containing images 

1 1 interposed over and eclipsing the main image on screen 2. In one embodiment, the 

12 images in regions 4a to 4d can be generated using a 3D graphics chip within the video 

13 display system. Circuitry for providing the images in regions 4a to 4d can be as 

14 described in U.S. Patent Application 09/344,442, filed June 25, 1999. (The '442 

15 application discloses means for binding 2D images to a planar surface using a 3D 

16 graphics pipeline. The 4 442 application is incorporated herein by reference.) 

17 In a version of the third embodiment, one can adjust the images in regions 4a to 

18 4d from being completely opaque to completely transparent, or somewhere in between. 

19 In other words, when the images in regions 4a to 4d are completely opaque, any 

20 underlying image is completely eclipsed. By rendering regions 4a to 4d more 

2 1 transparent, one can perceive a "ghost image" of the information displayed in regions 4a 

22 to 4d along with the underlying image. By rendering regions 4a to 4d completely 

23 transparent, one simply sees the underlying image in regions 4a to 4d of screen 2. 
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1 In a fourth embodiment, regions 4a to 4d display images that periodically change. 

2 For example, for thirty seconds, a first thumbnail still or video image appears in region 

3 4a, and thereafter, a second image appears in region 4a. After another thirty seconds 

4 elapses, a third image appears in region 4a. The link associated with region 4a can 

5 change at the same time the image changes. Alternatively, in other embodiments, the 

6 link remains unchanged. In one embodiment, invoking the link associated with region 4a 

7 alters the image shown in regions 4b to 4d. In another embodiment, invoking the link 

8 associated with one region does not affect the images shown in the other active regions. 

9 In a fifth embodiment, the size and/or appearance of regions 4a to 4d can be 

10 changed, e.g. by actuating appropriate buttons on controller 10. Thus, regions 4a to 4d 

1 1 can be made smaller or larger. 

12 In one embodiment, the links associated with regions 4a to 4d are related to the 

13 content displayed in region 6. (In this embodiment, the template comprising regions 4a 

14 to 4d is typically provided by the content provider of the image shown in region 6.) For 

15 example, if region 6 displays a sporting event, regions 4a to 4d might coirespond to links 

16 for displaying information about that or related sporting events. Thus, if the sporting 

17 event is a baseball game, regions 4a to 4d might correspond to links concerning statistics 

1 8 pertaining to the teams or players. In addition, the images depicted in regiosn 4a to 4d 

19 are related to the content displayed in region 6. However, in another embodiment, the 

20 images shown at regions 4a to 4d are unrelated to the content displayed in region 6. 

21 As mentioned above, regions 4a to 4d form a template. Suppose the image on 

22 screen 2 is annotated with links corresponding to active regions 12 and 14. Region 12 is 

23 located within region 6 and outside regions 4a to 4d, e.g. by using cursor 8 to click on 
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1 active region 12. Thus, a user can invoke the link associated with region 12 by using 

2 cursor 8 to click on region 12, even though the template is active. However, region 14 is 

3 located within region 4c. The template covers active area 14, and the link associated with 

4 area 14 cannot be invoked unless one removes the template (e.g. in a manner discussed 

5 below). Thus, if one moves cursor 8 to area 14 in an effort to invoke the link associated 

6 with area 14, one will only succeed in invoking the link associated with region 4c. 

7 As mentioned above, in one embodiment, the image on screen 2 is a video image. 

8 This image can be annotated with links in the manner described in the above- 

9 incorporated patent applications. In other words, portions of a video image are associated 

10 with interactive links. By placing the template over the video image, any links 

1 1 underneath regions 4a to 4d are effectively masked. 

12 In another embodiment, the image on screen 2 is a web page. Web pages 

13 typically include active areas for invoking links to other web pages. If screen 2 displays 

14 a web page, those links appearing within region 6 can be accessed by a user, and invoked 

15 in the normal manner. However, those links within regions 4a to 4d are effectively 

16 masked, and cannot be invoked by a user unless the template is removed. 

17 The template itself is typically stored in a memory device associated with the 

18 video system of which screen 2 is part. This memory device can be a semiconductor 

19 memory such as a RAM, a ROM, an EPROM, a disk drive, tape drive, or other memory 

20 device. This memory stores the location of the active regions 4a to 4d, and the links (e.g. 

21 URLs) with which regions 4a to 4d are associated. In one embodiment, the template (e.g. 

22 the links associated with active regions 4a to 4d) is determined by the user. In other 

23 words, a person viewing screen 2 can determine the links Hiat are associated with regions 
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1 4a to 4d, e.g. by inputting appropriate link address information into a memory, e.g. with 

2 an alphanumeric keypad coupled to video system 1 . (The keypad can be part of 

3 controller 10 or some other structure coupled to system 1 .) The user can activate or 

4 deactivate the template. 

5 In another embodiment, the template can originate from the same source as the 

6 video information displayed on screen 2. For example, if the program displayed on 

7 screen 2 originates from a cable TV source, the links associated with active regions 4a to 

8 4d also originate from that source. (Optionally, the locations of active regions 4a to 4d 

9 within screen 2 can also originate from that source.) 

10 In another embodiment, the links can originate from a source that is different from 

1 1 the origin of the image displayed on screen 2. For example, the image on screen 2 can be 

12 a conventional television program received from a radio antenna, whereas the links can 

13 be obtained via the internet. Optionally, the location of regions 4a to 4d can also be 

14 received from the internet. 

15 In one embodiment, a user can activate or deactivate the template, e.g. by pressing 

16 an appropriate control button on remote control device 10. If deactivated, regions 4a to 

17 4d no longer serve as active regions, and any active regions previously masked by 

18 regions 4a to 4d become unmasked. Thus, active region 14, previously masked by region 

19 4b, can then be clicked on, and the link associated with active region 14 can be invoked. 

20 When deactivated, any visual indication of the location of regions 4a to 4d is removed. 

21 Thus, if regions 4a to 4d previously had a different appearance, when the template is 

22 deactivated, regions 4a to 4d of screen 2 no longer have a different appearance. If, prior 

23 to deactivation, cursor 8 had a different appearance when positioned in regions 4a to 4d, 
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1 when deactivated, cursor 8 no longer takes on a different appearance when positioned in 

2 regions 4a to 4d. Regions 4a to 4d are typically activated or deactivated together as a 

3 group, e.g. by actuating the above-mentioned control buttons. In another embodiment, 

4 regions 4a to 4d can be individually activated or deactivated. 

5 A user can also reactivate the template, e.g. by actuating an appropriate control 

6 button on remote control device 1 0. Alternatively, a user can activate a different 

7 template, e.g. by actuating an appropriate control button on remote control device 10. 

8 This different template can have active regions having shapes and locations that are 

9 different from regions 4a to 4d. Also, the links associated with this different template can 

10 be to data sources other than the links associated with regions 4a to 4d. 

1 1 A user can also place one template over one or more other templates. For 

12 example, referring to Fig, 2, by actuating an appropriate button on controller 10, a user 

13 can activate a template having regions 16a to 16d. As can be seen, region 16a entirely 

14 covers and masks region 4a. Region 16b is elliptical, and only masks a portion of region 

15 4b. Thus, portions of region 4b are still accessible. Region 16c is trapezoidal, but still 

16 covers and masks all of region 4c. A user can deactivate the template corresponding to 

17 regions 16a to 16d if so desired by actuating an appropriate button on controller 10. 

1 8 In one embodiment, the various templates can be activate, deactivated, and placed 

19 over one another by an external source, e.g. the source providing the image in region 6 of 

20 screen 2. As indicated above, this image can come from any of a number of places, e.g. a 

2 1 video broadcast using radio waves, optical cable or electrical cable. This image can 

22 originate from other sources as well, e.g. the internet Thus, the provider of this image 

23 can determine which templates should be associated with the image at any given time. 
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1 In one embodiment, a template can be activated for only a predetermined time and 

2 then deactivated. Thus, if the image is a quiz show, the active regions can correspond to 

3 the answers to a question being asked on the quiz show. The active template might 

4 remain active for only so long as the question was pending. 

5 In one embodiment, each region 4a to 4d represents an independent link. In 

6 another embodiment, one can actuate regions 4a to 4d in different orders to obtain 

7 different results. For example, if one clicked on regions 4a, 4b, 4a, 4d, that might 

8 constitute a "code" for causing a certain event to occur, or for invoking a certain link. 

9 Alternatively, if one clicked on a different sequence of regions, that might cause a 

10 different event to occur, or invoke a different link. Templates could be added or 

1 1 removed, depending upon this order. Alternatively, templates could be locked in place or 

1 2 locked out, depending upon this order. (This capability of requiring active areas 4a to 4d 

13 to be actuated in a certain order could be used for security purposes, e.g. for permitting or 

14 forbidding a user to access certain information, video images or web pages. 

1 5 Alternatively, this capability might also be part of a game, e.g. a user would have to 

16 figure out certain clues in order to figure out the correct order in which to actuate 

17 regions.) 

18 One embodiment of our invention can be practiced using a PC having the 

19 following: 

20 1 . A CPU such as a Celeron or Pentium, e.g. as manufactured by Intel, or a K6/K7 

2 1 processor, e.g. as manufactured by Advanced Micro Devices. 

22 2 . 24 MB of memory or greater. 
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1 3. The operating system can be Windows 95, Windows 98, WinCE, Win2000 ? or 

2 any other operating system that supports Direct X, Direct 3D and/or Direct Draw. 

3 These packages can be used to apply images to regions 4a to 4d. The Windows 

4 operating system includes a standardized platform called Direct X for Windows. 

5 Fig. 3 is a block diagram of a computer system 50 for performing a method in 


6 accordance with our invention. Referring to Fig. 3, system 50 comprises a CPU 52, e.g. a 

7 Pentium II class CPU, comprising a cache memory 52a, a core 52b and an internal bus 

8 52c for facilitating communication between core 52b and cache 52a. Core 52b 

9 communicates via a CPU bus 54 to a system controller 56. System controller 56 

10 communicates with the system memory 58 via a memory bus 60. System memory 58 

1 1 includes system memory. 

12 Also included in system 50 is a PCI bus 62 for facilitating communication 

13 between system controller 56 and I/O devices 64, 66 and disk drive 68. I/O device 64 

14 can be any type of I/O device, e.g. a modem or telephone for communicating with a 

15 telephone line. In one embodiment, I/O device 66 is a video capture card with a driver. 

16 The video capture card can be coupled to receive a video program from an antenna, a 

17 cable or optical fiber, a VCR, a video disk, or other video signal source. Data from the 

18 video capture card is either loaded by DMA (direct memory access) or CPU 52 into a 

19 frame buffer, typically within main memory 58. However, the frame buffer may be in 

20 other memories within system 50. 

21 Graphics controller 70 uses its own local memory 74 to generate and store pixel 

22 arrays to be displayed on a video display unit 76. 
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1 It is emphasized that system 50 is only one example of a system that performs a 

2 method in accordance with our invention. Other hardware can be used as well 

3 Stored within a memory within video display system 1 are the locations on screen 

4 2 of regions 4a to 4d. CPU 52 (or other logic hardware within system 1) determines 

5 whether cursor 8 has been moved to one of regions 4a to 4d, i.e. by comparing the 

6 position of cursor 8 to the values stored in memory corresponding to the location of 

7 regions 4. (This memory can be memory 58 or another memory within the system.) 

8 As mentioned above, the image displayed on screen 2 can be annotated with 

9 active areas. The locations of these active areas (and the links associated with the active 

10 areas) are also stored in the above-mentioned memory (e.g. memory 58). Microprocessor 

1 1 52 (or the above-mentioned other hardware) also compares the position of cursor 8 with 

12 the locations of these other active areas. However, microprocessor 52 (or the other 

13 hardware within the system) also ascertains whether these other active areas are masked 

14 by the template (e.g. regions 4a to 4d). If the cursor is moved over an active area which 

15 is masked by region 4a, for example, microprocessor 52 ascertains that the link 

1 6 associated with region 4a is to be invoked. 


17 Memory 58 can store numerous templates, each of which can be activated or 

18 deactivated, e.g. by controller 10. These templates can also be placed over one another. 

19 White the invention has been described with respect to specific embodiments, 

20 those skilled in the art will appreciate that changes can be made in form ;uid detail 

21 without departing from the spirit and scope of the invention. For example, any type of 

22 display screen can be used in conjunction with the invention. For example, projection 

23 video display screens, LCDs, CRTs or other types of display devices can be used. 
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1 Similarly, although the drawings display a screen depicting four active regions, different 

2 numbers of active regions can be used. 

3 Although the template of Fig. 1 permits active areas within region 6 to be 

4 activated, in other embodiments, only the active areas of the active template itself can be 

5 activated. 

6 As mentioned above, in one embodiment video system 1 displays various images 

7 in regions 4a to 4d. In one embodiment, these images can be applied to regions 4a to 4d 

8 using a method discussed in U.S. Patent Application 09/344,442. Accordingly, ail such 

9 changes come within the invention. 
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